Real-time fail-over recovery for a media area network

ABSTRACT

A media area network includes a storage system having at least one storage device for storing digitized information. A host bus adapter provides a link between the storage system and a host system that provides overall control of the media area network. Within the host bus adapter, a lower-level port driver monitors communications between the storage system and the host bus adapter. In the event of a communications failure, the lower-level port driver initiates switching from a failed port to an alternative port, thereby achieving fail-over recovery. Allocating the responsibility for fail-over recovery to the lower-level port driver assures timely handling of port failures, thereby reducing potential latency delays.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority under 35U.S.C. 119(e) to U.S.Provisional patent application Ser. No. 60/400,635 filed Aug. 2, 2002,the teachings of which are incorporated herein.

TECHNICAL FIELD

This invention relates to a technique for achieving fail-over recoveryof storage devices in a media area network.

BACKGROUND ART

Within the broadcast industry, there exist Media Area Networks (MANs)that comprise a host system, in the form of a central processor thatexecutes a non-real time operating system. A host bus adapter links thehost system to a storage system that includes one or more storagedevices. Each device can take the form of a stand-alone disk or aRedundant Array of Inexpensive Disks (RAID). In practice, each storagedevice holds digitized video accessible for editing and/or broadcast. Toassure reliability, all components within the MAN are fault tolerant andhave redundant features in an effort to offer real time recovery in theevent of a fault. Such real time recovery becomes especially criticalwhen the video stored in one or more of the storage devices of thestorage system undergoes live transmission.

When a fault occurs in a MAN, the location of the fault can affect thetime required for recovery. For example, consider a fault associatedwith a port assigned to a storage device. Upon the occurrence of such afault, an error signal propagates into a fibre channel fabric thatcarries the error signal to the host bus adapter. The host bus adaptertypically has the capability the switching between the failed port andan alternative port to recover from the fault. Unfortunately,present-day host bus adapters do not inherently support the real-timerequirements of a media area network. Existing host bus adapters usuallyintroduce significant latencies. Delays of as much as 10 seconds canoccur between receipt of an error and the switching between ports. Suchdelays impose significant difficulties. Some manufacturers of host busadapters now provide fail-over recovery software that manages portfailures. Unfortunately, such software has not proven to be eithertransparent or seamless. Empiric testing has revealed that such softwareincurs latency delays as much as 15 seconds.

Thus, there is need for a technique for providing near real timerecovery of faults in a MAN.

BRIEF SUMMARY OF THE INVENTION

Briefly, in accordance with present principles, a Media Area Network(MAN) includes a host system, a storage system having at least onestorage device, and a host bus adapter linking the host system to thestorage system. Within the host bus adapter, a port driver monitors forthe presence of one or more error signals generated by the storagesystem upon the occurrence of an error. In response to an error signal,the port driver automatically initiates switching between a failed portand an alternative port to accomplish fail-over recovery. Allocating theresponsibility for fail-over recovery to the port driver assures timelyhandling of port failures, thereby reducing potential latency delays.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block schematic diagram of a MAN that accomplishesreal-time fail-over recovery in accordance with the present principles;

FIG. 2 depicts a flow chart representation of the steps executed toperform the task of 25 servicing an interrupt generated by storagedevice in the MAN of FIG. 1; and

FIG. 3 depicts a flow chart representation of the steps executed toperform real time fail-over recovery after detecting certain types oferrors during the task of servicing an interrupt depicted in FIG. 2.

DETAILED DESCRIPTION

FIG. 1 depicts a block schematic diagram of an illustrative embodimentof a Media Area Network (MAN) 10 that includes a host system 12 linkedby a host bus adapter 14 to a storage system 16. The storage system 16includes one or more storage devices, exemplified by device 18. Eachstorage device 18 can take to form of an individual device, or aRedundant Array of Inexpensive Disks (RAID). Each storage device 18 hasthe capacity to store large volumes of digitized information, such asdigitized video, either in compressed or uncompressed form. Within thestorage system 16, a fibre channel fabric 20 couples each storage device18 to the host bus adapter 14. The fibre channel fabric 20 typicallytakes the form of a one or more conventional fibre channel switches andassociated links (not shown).

The host bus adapter 14 provides a switchable path between the hostsystem 12 and the storage system 16. To that end, the host bus adapter14 includes a real-time kernel 22 in the form of a processor running areal time operating system, such as the VxWorks™ operating systemavailable from Wind River Systems, Inc., Alameda, Calif., although otherreal-time operating systems exist and can readily be employed. Thereal-time kernel 22 controls a lower-level Small Computer SystemsInterface (SCSI) interface port driver 24 that provides real-timefail-over recovery functionality in accordance with the presentprinciples. In particular, the lower-level port driver 24 includes logic(either in the form of dedicated circuitry or a programmable processor)for monitoring the status of individual ports and associated links (bothnot shown) that carry information to and from the storage system 16. Toassure redundancy, each storage device 18 maintains a connection to thehost bus adapter 14 through dual links and dual ports. One of the portsand its associated link serves as an alternate while the other port andassociated link remain active. In the event of a failure (e.g., thefailure of a previously active port and/or its associated link), thelower-level port driver 24 switches to the alternate port (and itsassociated link) to achieve fail-over recovery. As described in greaterdetail with respect to FIGS. 2 and 3, the lower-level port driver 24thus performs the decision-making associated with the port switching (aswell as the decision making concerning activating a redundant storagedevice and/or device controller). Accordingly, the lower level portcontroller 24 relieves the host system 12 of this responsibility, whichreduces latency delays. The lower-level port driver 24 also serves tofacilitate communications for SCSI I/O traffic through the fibre channelfabric 20.

In the illustrated embodiment of the MAN 10 in FIG. 1, the host busadapter 14 connects to the fibre channel fabric 20 via dual connections(i.e. two links and two ports per channel). The storage system 18likewise connects to the fibre channel fabric 20 via two connections perRAID chassis. In this way, either of the two host ports can communicatewith either of two RAID controllers (not shown) per RAID chassis. Thisallows for independent fail-over between the ports and the two RAIDcontrollers. Each host port can use either RAID controller in a RAIDchassis. In the event of a failure, host port switching can occurwithout switching RAID controllers and RAID controller switching canoccur without switching host ports.

The host system 12 provides overall control of the MAN 10 via a non-realtime kernel 26 that takes the form of a processor executing a non-realtime operating system, such the Windows® operating system from MicrosoftCorporation, Redmond, Wash., the Solaris® operating system from SunMicrosystems, Santa Clara, Calif., or the Linux operating system. Thenon-real time kernel 26 communicates with the host bus adapter 14 via amessaging technique, rather than a direct connection with each storagedevice 18, to manage the communication of information between thestorage system 16 and the host system 12.

FIG. 2 illustrates a flow-chart that depicts the steps of a methodexecuted by the lower-level port driver 24 of FIG. 1 to accomplish thetask of servicing an interrupt generated by 15 the storage device 18 inthe storage system 16 of FIG. 1. The task of servicing an interruptcommences upon execution of step 100 during which the lower-level portdriver 24 checks whether the storage device 18 of FIG. 1 completed acommand in a normal manner. If so, then the lower-level port driver 24will advise the host system 12 of FIG. 1 of the successful completion ofthat command during step 110 of FIG. 2. Following unsuccessful executionof a storage system command during step 100, a check occurs during step120 whether the error is correctable. In other words, the lower-levelport driver 24 determines whether the error that occurred can becorrected by switching to an alternate port or controller. Upondetermining that no corrective action exists, the host system 12 of FIG.1 receives a notification to that effect during step 130 of FIG. 2. Inthe event of a correctable error, the lower-level port driver 24proceeds to mark the port (not shown) associated with the storage devicethat generated the error as inactive during step 140 of FIG. 2.Thereafter, the lower-level port driver 24 schedules the task offail-over recovery (i.e., the task of selecting an alternative port)during step 150 of FIG. 2.

FIG. 3 illustrates a flow chart that depicts the steps of the task offail-over recovery performed by the lower-level port driver 24 ofFIG. 1. The task of fail-over recovery commences upon execution of step200 of FIG. 3 during which the lower-level port driver 24 waits for asignal from the interrupt task of FIG. 2 indicating that the task offail-over recovery should occur. Upon finding that the task of fail-overrecovery has now become active, the lower-level port driver 24 of FIG. 1places all requests from the inactive (i.e., failed) port in a queueduring step 210 of FIG. 3. Thereafter, the lower-level port driver 24cancels all outstanding requests from the original, but now inactiveport during step 215 of FIG. 3, typically by way of a Third PartyProcess Log Out (TPPLO) command. Next, a check is made during step 220of FIG. 3 whether the TPPLO command failed. Upon detecting a failure ofthe TPPLO command during step 220, the lower-level port driver 24 ofFIG. 1 makes an inference during step 225 that the controller (notshown) associated with the storage device 18, (typically a RAIDcontroller) failed or the path associated with the controller failed.Under such circumstances, the lower-level port driver 24 of FIG. 1 willinitiate recovery by actuating a redundant RAID controller.

Following step 225 (or step 220 when the TPPLO command did not fail),the lower-level port controller 24 of FIG. 1 completes (i.e., “cleansup”) any existing Test Unit Ready (TUR) responses from any of thestorage devices 18 during step 230 of FIG. 3. Finally, the lower-levelport controller 24 begins issuing commands through the newly activatedalternate port during step 240, including commands previously queued forretry during step 210. Thereafter, program execution branches back tostep 200 to await the recovery task.

The foregoing describes a technique for achieving fail-over recovery ofstorage devices in a media area network by having a lower-level portdriver 24 monitor for a failed (inactive) port and then switch to analternative port to effect recovery.

1. A media area network comprising: a storage system including at leastone storage device for storing digitized information; a host system forproviding overall control of the media area network; and a host busadapter for providing a link between the host system and the storagesystem, the host bus adapter having a lower-level port driver thatincludes: means for monitoring communications between the storage systemand the host bus adapter through an active port, and means for switchingto an alternative port in real time, thereby achieving fail-overrecovery in the event of a communications failure.
 2. The media areanetwork according to claim 1 wherein the monitoring means furthercomprises means for determining whether the storage system successfullycompleted at least one command.
 3. The media area network according toclaim 2 wherein the monitoring means further comprises means fordetermining whether unsuccessful completion of the at least one commandcan be corrected by fail-over recovery.
 4. The media area networkaccording to claim 3 wherein the switching means further comprises meansfor scheduling fail-over recovery upon determination that unsuccessfulcompletion of the at least one command can be corrected by fail-overrecovery.
 5. The media area network according to claim 4 wherein themeans for scheduling fail-over recovery further comprises: means forqueuing requests from an original port that failed to an alternativeport; means for canceling all outstanding requests on the original port;and means for issuing at least one command via the alternate port.
 6. Amethod for achieving fail-over recovery in a media area network having astorage system with at least one storage device for storing digitizedinformation, a host system for providing overall control of the mediaarea network; and a host bus adapter for providing a link between thehost system and the storage system, the method comprising the steps ofmonitoring, at a lower-level port driver in the host bus adapter,communication status between the storage system and the host busadapter, and in the event of a failure; initiating switching at thelower-level port driver to activate an alternative port, therebyachieving fail-over recovery.
 7. The method according to claim 6 whereinthe step of monitoring the communication status between the storagesystem and the host bus adapter further comprises the step ofdetermining whether the storage system successfully completed at leastone command.
 8. The method according to claim 7 further comprising thestep of determining whether unsuccessful completion of the at least onecommand can be corrected by fail-over recovery.
 9. The method accordingto claim 8 further comprising the step of scheduling fail-over recoveryupon a determination that unsuccessful completion of the at least onecommand can be corrected by fail-over recovery.
 10. The method accordingto claim 9 further comprising the steps of: queuing requests from anoriginal port that failed to an alternative port; canceling alloutstanding requests on the original port; and issuing at least onecommand via the alternate port.
 11. The method according to claim 10further comprising the step of checking whether cancellation of theoutstanding commands occurred, and if not then initiating fail-overrecovery of any failed storage system controller.